Goal¶
Using the data collected from existing customers, build a model that will help the marketing team identify potential customers who are relatively more likely to subscribe term deposit and thus increase their hit ratio
git repo:
- Exploratory Data Analysis
- Prepare Data for Analytics
- Create Ensemble Model
- Logistics Regression using statsmodel
- Logistic Regression from Sklearn
- Build Decision Tree Model
- Decision Tree reducing over fitting (Regularization)
- Ensemble Learning - Bagging
- Ensemble Learning - AdaBoosting
- Ensemble Learning - GradientBoost
- Ensemble RandomForest Classifier
- compare models and conclusion
Suppress the warnings¶
In [1]:
import warnings
warnings.filterwarnings('ignore')
Import libraries¶
In [2]:
# data read and structuring
import pandas as pd
import numpy as np
# visualization
import seaborn as sns
import matplotlib.pyplot as plt
# model building
from sklearn.linear_model import LogisticRegression
# data preparing
from sklearn.model_selection import train_test_split
# check error values
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
# custom display output
from IPython.display import display, HTML
import scipy.stats as stats
# scalars to normalize and impute
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import PowerTransformer
from sklearn.preprocessing import QuantileTransformer
from sklearn.preprocessing import RobustScaler
# Classifiers
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import GradientBoostingClassifier
# Scoring mertics
from sklearn import metrics
from sklearn.metrics import accuracy_score,f1_score,recall_score,precision_score, confusion_matrix
# Imaging tools
from IPython.display import Image
from sklearn import tree
from os import system
%matplotlib inline
sns.set(color_codes=True)
Exploratory Data Analysis ¶
In [3]:
df = pd.read_csv('bank-full.csv')
In [4]:
df.head()
Out[4]:
In [5]:
df.shape
Out[5]:
In [6]:
df.dtypes
Out[6]:
Univariate Analysis ¶
In [7]:
df.describe().transpose()
Out[7]:
In [8]:
df.describe(include='all').transpose()
Out[8]:
from the data 5 point summary below,¶
from the initial look at 5-point summary, looks like we need to include all features to decide the outcome (term deposit)
'default', 'housing', 'loan' are discrete binary categorical value and 5 -point summary does not apply but are independent features that will influence the outcome (target: term deposit)
'job', 'marital', 'education', 'contact', 'day', 'month', 'poutcome' are multi-class categorical feature that may influence the outcome
'day' need to be treated as a categorical feature
Contineous features that impact the outcomes are 'age', 'balance', 'duration', 'campaign', 'pdays', 'previous'
target or dependent or outcome variable is 'target'. Since this is a discrete binary categorical value, we will apply 'Classifications'
'balance' have -ve values and shall be treated
'pdays' -1 value will be converted to 999 to have the weighted contineous value
In [9]:
# check null or missing values
df.isnull().sum()
Out[9]:
The data set got 0 missing cells.¶
In [10]:
# check duplicate observations
data_dup = df[df.duplicated(keep="last")]
data_dup
Out[10]:
In [11]:
# check data types and null missing values
df.info()
In [12]:
# from the 5-point summary, there are some -ve balance. Treat and correct it.
df['balance'] = df['balance'].abs()
In [13]:
# Treat the -1 value in pdays, to get a predictable distribution and inform
# that the customers were really not been contacted for long, replace all -1 with 999
df.pdays[df.pdays == -1] = 999
In [14]:
# create a list of categorical and contineous features
categorical_cols = ['default', 'housing', 'loan', 'job', 'marital',
'education', 'contact', 'day', 'month', 'poutcome']
contineous_cols = ['age', 'balance', 'duration', 'campaign', 'pdays', 'previous']
Value counts of all categorical columns.¶
In [15]:
# Value counts of the categorical features to find the data population
for col in categorical_cols:
print(df[col].value_counts())
print()
Percent of all categorical columns.¶
In [16]:
# Also find the percentage of value count of contineous features to determine the data population we need to process
for col in categorical_cols:
print(df[col].value_counts(normalize=True) * 100)
print()
Approximately 98% of the candidates are not defaulted¶
candidates having housing loans are slightly more than not having but fairly even¶
Approximately 84% of the customer have not taken personal loan¶
job categories are not even, people with blue-collar, management, technicians are almost 56% of the population¶
Around 60% of customers married¶
Around 51% of the customer have secondary education¶
64% of the customers were contacted on their cell phones¶
customers were contacted almost all days of the month, however less during the start and end of the month¶
Almost 55% of customer were contacted last during the mid of the year¶
Previous marketing compaign were almost 'unknown' and probably we can srop this feature since the data results is unknown¶
In [17]:
# check the spread of data to again determine the data population
df.nunique()
Out[17]:
In [18]:
# Term deposit count based on educational level
def independent_count_plot(independent):
plt.figure(figsize=(10,7))
plt.xticks(rotation=70)
plt.title('Count of ' + independent + ' feature', fontsize=20)
plt.xlabel(independent, fontsize=10)
plt.ylabel('Count', fontsize=10)
sns.countplot(x=independent, data=df)
plt.show()
# # count of a given independent feature against the dependent feature
# def count_plot(c):
# indep_vs_dep = pd.crosstab(df[independent], df[dependent])
# print("Count:")
# print (indep_vs_dep)
# print()
# print("Percent:")
# print (indep_vs_dep.div(indep_vs_dep.sum(1).astype(float), axis = 0) * 100)
# print()
# plt.figure(figsize=(10,7))
# plt.xticks(rotation=70)
# plt.title(dependent + ' count per ' + independent + ' feature', fontsize=20)
# sns.countplot(x=dependent, hue=independent, data=df);
# plt.show()
# indep_vs_dep.div(indep_vs_dep.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
# count of a given independent feature against the dependent feature
def count_plot(independent, dependent):
indep_vs_dep = pd.crosstab(df[independent], df[dependent])
print("Count:")
print (indep_vs_dep)
print()
print("Percent:")
print (indep_vs_dep.div(indep_vs_dep.sum(1).astype(float), axis = 0) * 100)
print()
plt.figure(figsize=(10,7))
plt.title(dependent + ' count per ' + independent + ' feature', fontsize=20)
sns.countplot(x=dependent, hue=independent, data=df);
plt.show()
indep_vs_dep.div(indep_vs_dep.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
def countplot(label):
plt.figure(figsize=(15,10))
Y = df[label]
total = len(Y)*1.
ax=sns.countplot(x=label, data=df)
for p in ax.patches:
ax.annotate('{:.1f}%'.format(100*p.get_height()/total), (p.get_x()+0.1, p.get_height()+5))
ax.yaxis.set_ticks(np.linspace(0, total, 11))
ax.set_yticklabels(map('{:.1f}%'.format, 100*ax.yaxis.get_majorticklocs()/total))
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
plt.show()
def countplot_withY(label, dependent):
plt.figure(figsize=(20,10))
Y = df[label]
total = len(Y)*1.
ax=sns.countplot(x=label, data=df, hue=dependent)
for p in ax.patches:
ax.annotate('{:.1f}%'.format(100*p.get_height()/total), (p.get_x()+0.1, p.get_height()+5))
ax.yaxis.set_ticks(np.linspace(0, total, 11))
ax.set_yticklabels(map('{:.1f}%'.format, 100*ax.yaxis.get_majorticklocs()/total))
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
plt.show()
def univariant_categorical(independent, dependent):
count_plot(independent, dependent)
countplot(independent)
countplot_withY(independent, dependent)
# independent_count_plot(independent)
# Univariant: Personal loan count based on educational level
# categorical_columns = ['Family', 'Education', 'Securities Account', 'CD Account', 'Online', 'CreditCard']
# for col in categorical_columns:
# univariant_categorical(col, 'Personal Loan')
In [19]:
for col in categorical_cols:
univariant_categorical(col, 'Target')
In [20]:
def box_independent_hue(independent):
plt.figure(figsize=(10,7))
sns.boxplot(data=df, x = 'Target', y = independent);
plt.show()
def box_independent(independent):
plt.figure(figsize=(10,7))
sns.boxplot(data=df, x = independent);
plt.show()
def dist_plot(independent, skip_dist_plot=False):
if skip_dist_plot:
return
plt.figure(figsize=(10,7))
sns.distplot(df[independent]);
def dist_plot_hue(independent):
plt.figure(figsize=(10,7))
# sns.countplot(x=independent, hue='Target', data=df);
plt.hist(df[independent], density=True)
def density_plot(independent, dependent):
sns.distplot(df[df[dependent]=='no'][independent],color='r',label=0)
sns.distplot(df[df[dependent]=='yes'][independent],color='g',label=1)
plt.legend()
plt.show()
def univariant_contineous(independent, dependent, skip_dist_plot=False):
# density_plot(independent, dependent)
box_independent(independent)
# box_independent_hue(independent)
dist_plot_hue(independent)
dist_plot(independent, skip_dist_plot)
In [21]:
univariant_contineous('age', 'Target')
In [22]:
univariant_contineous('balance', 'Target')
In [23]:
univariant_contineous('duration', 'Target')
In [24]:
univariant_contineous('campaign', 'Target')
In [25]:
df["previous"].unique()
Out[25]:
In [26]:
df["previous"].value_counts()
Out[26]:
In [27]:
df[df["Target"]=="yes"]["previous"].value_counts()
Out[27]:
In [28]:
df[df["Target"]=="no"]["previous"].value_counts()
Out[28]:
In [29]:
univariant_contineous('previous', 'Target', skip_dist_plot=True)
# box_independent('previous')
# box_independent_hue(independent)
# dist_plot_hue(independent)
# dist_plot(independent)
In [30]:
univariant_contineous('pdays', 'Target', skip_dist_plot=True)
Observation¶
Customer defaulted or not does not impact a lot to the term deposit account. As can be seen, 11% customers not defaulted have term deposit, whereas 6% of customers that are defaulted have term desposit account¶
16% of customer with no housing loan have term deposits, whereas 7% with housing loans have term deposit¶
12% of customer with no personal loan have term deposits, whereas 6% with personal loans have term deposit¶
28% of students, 22% of retired customers have term deposits, others job categories are fiarly distributed¶
From the above distribution we can see that most of the customers have jobs as "admin", "blue-collar" or "technician". One interesting thing to find out would be to see the distribution for each classes as well. For example, how many people who work as an admin have subscribed a term deposit.¶
From the above plot, we can see that the customers who have a job of admin have the highest rate of subscribing a term deposit, but they are also the highest when it comes to not subscribing. This is simply because we have more customers working as admin than any other profession. We can find out the odds or ratio of subscribing and not subscribing based on the profession, to find out which profession has the highest odds of subscribing given the data. At this point we are not sure if there is any correlation between job and target variable. Idea: If we find that odds of one profession subscribing is greater than other job, we can use the odds or log(odds) as a feature by replacing jobs field with the odds, instead of doing one hot encoding.¶
From the above plot, we can see that the customers who have a job of admin have the highest rate of subscribing a term deposit, but they are also the highest when it comes to not subscribing. This is simply because we have more customers working as admin than any other profession. We can find out the odds or ratio of subscribing and not subscribing based on the profession, to find out which profession has the highest odds of subscribing given the data. At this point we are not sure if there is any correlation between job and target variable. Idea: If we find that odds of one profession subscribing is greater than other job, we can use the odds or log(odds) as a feature by replacing jobs field with the odds, instead of doing one hot encoding.¶
Majority of the customers have a housing loan.¶
The day of the week seems to be irrelevent as we have the same amount of data for all the days of the week, and no:yes ratio is also almost same.¶
Multivariate Analysis ¶
In [32]:
def multivariant_contineous(independent, dependent):
density_plot(independent, dependent)
box_independent_hue(independent)
dist_plot_hue(independent)
In [33]:
df[df['pdays'] < 0]
Out[33]:
In [34]:
# Treat the -1 value in pdays, to get a predictable distribution and inform
# that the customers were really not been contacted for long, replace all -1 with 999
df.pdays[df.pdays == -1] = 999
In [35]:
df[df['pdays'] < 0].pdays
Out[35]:
Replace unknowns from 'contact' with 'cellular' and 'telephone' proportionately.¶
In [36]:
# Replace unknown values of 'contact' variable proportionately with cellular and telephone
contact_count = df.contact.value_counts()
cell_per = contact_count['cellular'] / (contact_count['cellular'] + contact_count['telephone'])
cell_cnt = int(round(contact_count['unknown'] * cell_per))
contact_ranges = df.contact.values.tolist()
for i in range(len(contact_ranges)):
if contact_ranges[i] == 'unknown':
if cell_cnt > 0:
contact_ranges[i] = 'cellular'
cell_cnt -= 1
else:
contact_ranges[i] = 'telephone'
df['contact'] = contact_ranges
print("unknown contact count: ", df[df.contact == 'unknown'].shape[0])
Treat unknown jobs¶
In [37]:
# Let's look at cross tabulation between job and education
pd.crosstab(df['education'], df['job'])
Out[37]:
In [38]:
# Customers with primary and secondary education have more blue-collar jobs; while customers with
# secondary education have more management jobs
df.loc[(df.job == 'unknown') &
((df.education =='primary') | (df.education =='secondary')), 'job'] = 'blue-collar'
df.loc[(df.job == 'unknown') & (df.education == 'tertiary'), 'job'] = 'management'
print("Number of customers left with unknown jobs: ", df[df.job == 'unknown'].shape[0])
In [39]:
# Let's look at the ages for different job categories.
ja_df = df.groupby('job').age.min().to_frame().T
ja_df = ja_df.append(df.groupby('job').age.mean().to_frame().T, ignore_index=True)
ja_df = ja_df.append(df.groupby('job').age.max().to_frame().T, ignore_index=True)
ja_df.index = ['min. age', 'avg. age', 'max. age']
ja_df
Out[39]:
In [40]:
# For customers who have unknown education and job values, we can try infering their job based on their ages
# Let's mark all customers above 60 as retired
df.loc[(df.job == 'unknown') & (df.age > 60), 'job'] = 'retired'
# Let's mark all customers between age 15 and 25 as students
df.loc[(df.job == 'unknown') & ((df.age > 15) & (df.age <= 25)), 'job'] = 'student'
# Based on distribution of age alone it's difficult to further infer job values.
# Let's mark the remaining customers (about 115) as unemployed for simplicity.
df.loc[df.job == 'unknown', 'job'] = 'unemployed'
print("unknown jobs count: ", df[df.job == 'unknown'].shape[0])
Treat unknown education¶
In [41]:
ea_df = df.groupby('education').age.min().to_frame().T
ea_df = ea_df.append(df.groupby('education').age.mean().to_frame().T, ignore_index=True)
ea_df = ea_df.append(df.groupby('education').age.max().to_frame().T, ignore_index=True)
ea_df.index = ['min. age', 'avg. age', 'max. age']
ea_df
Out[41]:
In [42]:
# Get popular education levels per job category
edu_per_job = df.groupby('job').apply(lambda x: x['education'].value_counts().index[0])
edu_job_dict = {x:y for (x,y) in zip(edu_per_job.index, edu_per_job.values)}
# Replace unknowns from education with corresponding popular education level for that job
df['education'] = df.apply(lambda x : edu_job_dict[x['job']] if x['education'] == 'unknown'
else x['education'], axis=1)
print("unknown education levels: ", df[df.education == 'unknown'].shape[0])
poutcome have a relation with pdays but not strong enough to build an imputation against it¶
In [43]:
cont_cols = ['age', 'balance', 'duration', 'campaign', 'previous', 'pdays']
for col in cont_cols:
print()
print (col, ':', df[col].value_counts())
In [44]:
cont_cols = ['age', 'balance', 'duration', 'campaign', 'previous', 'pdays']
for col in cont_cols:
try:
multivariant_contineous(col, 'Target')
except:
print('Cannot plot for ', col)
In [45]:
plt.figure(figsize=(10,8))
try:
sns.distplot(df[df["Target"]=="yes"]["previous"])
sns.distplot(df[df["Target"]=="no"]["previous"])
except:
print()
plt.show()
Obervation on the above chart¶
From the above boxplot we know that for both the customers that subscibed or didn't subscribe a term deposit, has a median age of around 38-40. And the boxplot for both the classes overlap quite a lot, which means that age isn't necessarily a good indicator for which customer will subscribe and which customer will not.¶
As we can see in the above distribution also, that most of the customers are in the age range of 30-40.¶
the duration (last contact duration) of a customer can be useful for predicting the target variable. It is expected because it is already mentioned in the data overview that this field highely affects the target variable and should only be used for benchmark purposes. This seems like a powerlaw distribution where most the values are very low and very few have high values.¶
Most of the 'pdays' values are 999, which means that the most of the customers have never been contacted before.¶
Quite a number of outliers in all contineous features, we will treat them using Robust Scaler¶
In [46]:
df = df.replace({'Target': {'yes': 1, 'no': 0}})
In [47]:
pair_cols = contineous_cols + ['Target']
sns.pairplot(df[pair_cols], hue='Target', diag_kind='hist');
In [48]:
df.corr()
Out[48]:
In [49]:
plt.figure(figsize=(20,15))
sns.heatmap(df.corr(), annot=True);
A fair correlation between pdays and previous, campaign and pdays and day¶
In [50]:
def average_contineous_plot(independent, dependent):
df.groupby(dependent)[independent].mean().plot(kind='bar')
plt.title('Average ' + independent + ' effect on ' + dependent, fontsize=20)
plt.show()
print()
for col in contineous_cols:
average_contineous_plot(col, 'Target')
In [51]:
df.columns
Out[51]:
In [52]:
contineous_cols
Out[52]:
In [53]:
# Let's look at the skewness of data
skew_df = pd.DataFrame({'Skewness' : [stats.skew(df.age),
stats.skew(df.balance),
stats.skew(df.duration),
stats.skew(df.campaign),
stats.skew(df.pdays),
stats.skew(df.previous)]},
index=['Age','balance','duration','campaign','pdays', 'previous'])
skew_df
Out[53]:
Prepare the data for analytics ¶
Ensure the attribute types are correct ¶
In [54]:
df.info()
Dealing with incorrect values¶
We have already treated -ve balance and pdays during EDA¶
In [55]:
df['balance'] = df['balance'].abs()
In [56]:
df[df.balance < 0]
Out[56]:
Obervation: no -ve balance¶
Dealing with missing values¶
From the above basic info of each feature, we know that there are no missing values in this dataset.¶
Dealing with duplicate observations¶
In [57]:
data_dup = df[df.duplicated(keep="last")]
data_dup
Out[57]:
No duplicates¶
Treat the outliers using Robust scalar¶
In [58]:
data_processed = df.copy()
In [59]:
# treat the outliers using RobustScalar
from sklearn import preprocessing
scaler = preprocessing.RobustScaler()
# Get list of independent variables to scale
variables_to_scale = ['age', 'balance', 'duration', 'campaign', 'previous', 'pdays']
data_processed[variables_to_scale] = scaler.fit_transform(data_processed[variables_to_scale])
data_processed
Out[59]:
One-hot encoding the categorical indpendent feature¶
In [60]:
categorical_feature_mask = df.dtypes==object
# filter categorical columns using mask and turn it into a list
categorical_cols = df.columns[categorical_feature_mask].tolist()
In [61]:
categorical_cols
Out[61]:
In [62]:
categorical_cols.append('day')
In [63]:
categorical_cols
Out[63]:
In [64]:
for col in categorical_cols:
data_processed[col] = data_processed[col].astype('category')
In [65]:
data_processed.dtypes
Out[65]:
In [66]:
data_processed = pd.get_dummies(data_processed, drop_first=True)
# data_processed.rename(columns={'Target_yes': 'Target'}, inplace=True)
Create training set and test set in ratio of 70:30 ¶
In [67]:
from sklearn.model_selection import train_test_split
pdata = data_processed.copy()
pdata
X = pdata.drop('Target',axis=1) # Predictor feature columns
y = pdata['Target'] # Predicted class (1=True, 0=False) (1 X m)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
X_train.head()
Out[67]:
In [68]:
X_train.columns
Out[68]:
Logistics Regression using statsmodel ¶
In [69]:
## Helping function to calculate the scores
def print_model_score(model, y_predict):
print("Training accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
print()
print('Confusion Matrix')
print(draw_cm(y_test,y_predict))
print()
print("Recall:",recall_score(y_test,y_predict))
print()
print("Precision:",precision_score(y_test,y_predict))
print()
print("F1 Score:",f1_score(y_test,y_predict))
print()
print("Roc Auc Score:",roc_auc_score(y_test,y_predict))
Model building using Logistic Regression from Sklearn¶
We will use the sklearn library to build the model and make predictions
In [70]:
#Build the logistic regression model
import statsmodels.api as sm
logit = sm.Logit(y_train, sm.add_constant(X_train))
lg = logit.fit()
In [71]:
#Summary of logistic regression
from scipy import stats
stats.chisqprob = lambda chisq, df: stats.chi2.sf(chisq, df)
print(lg.summary())
In [72]:
#Calculate Odds Ratio, probability
##create a data frame to collate Odds ratio, probability and p-value of the coef
lgcoef = pd.DataFrame(lg.params, columns=['coef'])
lgcoef.loc[:, "Odds_ratio"] = np.exp(lgcoef.coef)
lgcoef['probability'] = lgcoef['Odds_ratio']/(1+lgcoef['Odds_ratio'])
lgcoef['pval']=lg.pvalues
pd.options.display.float_format = '{:.2f}'.format
In [73]:
# FIlter by significant p-value (pval <0.1) and sort descending by Odds ratio
lgcoef = lgcoef.sort_values(by="Odds_ratio", ascending=False)
pval_filter = lgcoef['pval']<=0.1
lgcoef[pval_filter]
Out[73]:
Observation¶
- most of the independent features are contributing to the result
Model building using Logistic Regression from Sklearn ¶
We will use the sklearn library to build the model and make predictions
In [74]:
# from sklearn import metrics
# from sklearn.linear_model import LogisticRegression
# from sklearn.metrics import confusion_matrix, recall_score, precision_score, f1_score, roc_auc_score,accuracy_score, classification_report
# # Fit the model on train
# lg_clf = LogisticRegression(random_state=42)
# lg_clf.fit(X_train, y_train)
In [75]:
col_types = [('classifier', str),
('train score', np.float128),
('test score', np.float128),
('recall score', np.float128),
('precision score', (np.float128)),
('F1 score', np.float128),
('roc auc score', np.float128)]
df_compare_model = pd.DataFrame({k: pd.Series(dtype=t) for k, t in col_types})
df_compare_model.style.format({'train score': '{:.6f}', 'test score': '{:.6f}', 'recall score': '{:.6f}',
'precision score': '{:.6f}', 'F1 score': '{:.6f}', 'roc auc score': '{:.6f}'})
Out[75]:
In [76]:
## function to get confusion matrix in a proper format
def draw_cm( actual, predicted):
cm = confusion_matrix( actual, predicted)
print (cm)
# print('The confusion matrix')
print('')
print ('True Positives (TP): we correctly predicted that they will take personal loans ', cm[1, 1])
print('')
print ('True Negatives (TN): we correctly predicted that they will not take personal loan ', cm[0, 0])
print('')
print ('False Positives (FP): we incorrectly predicted that they will take personal loan (a "Type I error") ', cm[0, 1], ' Falsely predict positive Type I error')
print('')
print ('False Negatives (FN): we incorrectly predicted that they will not take personal loans (a "Type II error")', cm[1, 0], ' Falsely predict negative Type II error')
sns.heatmap(cm, annot=True, fmt='.2f', xticklabels = [0,1] , yticklabels = [0,1] )
plt.ylabel('Observed')
plt.xlabel('Predicted')
plt.show()
In [77]:
from decimal import *
getcontext().prec = 7
def print_model_score(model, classifier):
global df_compare_model
predicted = model.predict(X_test)
print("Training accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
print()
print('Confusion Matrix')
print(draw_cm(y_test,predicted))
print()
print("Recall:",recall_score(y_test,predicted))
print()
print("Precision:",precision_score(y_test,predicted))
print()
print("F1 Score:",f1_score(y_test,predicted))
print()
print("Roc Auc Score:",roc_auc_score(y_test,predicted))
new_row = {'classifier':classifier,
'train score': Decimal(model.score(X_train,y_train)),
'test score': Decimal(model.score(X_test, y_test)),
'recall score': Decimal(recall_score(y_test,predicted)),
'precision score': Decimal(precision_score(y_test,predicted)),
'F1 score': Decimal(f1_score(y_test,predicted)),
'roc auc score': Decimal(roc_auc_score(y_test,predicted))}
# print (new_row)
df_compare_model = df_compare_model.append([new_row], ignore_index=True)
In [78]:
#AUC ROC curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
def print_model_roc(model):
logit_roc_auc = roc_auc_score(y_test, model.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.6f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()
In [79]:
from yellowbrick.classifier import ClassificationReport, ROCAUC
def visClassifierResults(model_w_parameters):
viz = ClassificationReport(model_w_parameters)
viz.fit(X_train, y_train)
viz.score(X_test, y_test)
viz.show()
roc = ROCAUC(model_w_parameters)
roc.fit(X_train, y_train)
roc.score(X_test, y_test)
roc.show()
In [80]:
def draw_tree(dtree_clf, xtr, ytr, dot_file, image_file, class_names, feature_names):
''' Method to draw decision tree.
dtree_clf - Decition Tree Classifier instance
xtr - train data
ytr - train labels
dot_file - dot file name with .dot extension
iamge_file - image file name with .png extension
class_names - labels
feature_names - list of independent features'''
dtree_clf.fit(xtr, ytr)
tree_file = open(dot_file,'w')
dot_data = tree.export_graphviz(dtree_clf,
out_file = tree_file,
feature_names = feature_names,
class_names = class_names,
filled=True, rounded=True, special_characters=True)
tree_file.close()
# Works only if "dot" command works on your machine
command = "dot -Tpng {0} -o {1}".format(dot_file, image_file)
retCode = system(command)
if retCode > 0:
print("Error while diplaying tree! error code: " + str(retCode))
else:
display(Image(image_file))
In [81]:
from sklearn import metrics
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, recall_score, precision_score, f1_score, roc_auc_score,accuracy_score, classification_report
# Fit the model on train
lg_clf = LogisticRegression(random_state=42)
lg_clf.fit(X_train, y_train)
Out[81]:
In [82]:
print_model_score(lg_clf, 'LogisticRegression')
df_compare_model
# df_compare_model.round(16)
# df_compare_model["train score"] = df_compare_model["train score"].apply(lambda x: (np.float128)(x*10000000000000000)/10000000000000000)
# df_compare_model.round(decimals=16)
# roundTwoDecimals = np.round(df_compare_model, decimals=6)
# roundTwoDecimals
Out[82]:
In [83]:
print_model_roc(lg_clf)
In [84]:
print(classification_report(y_test, lg_clf.predict(X_test)))
In [85]:
visClassifierResults(lg_clf)
Build Decision Tree Model ¶
We will build our model using the DecisionTreeClassifier function. Using default 'gini' criteria to split. Other option include 'entropy'.
In [86]:
dTree = DecisionTreeClassifier(criterion = 'gini', random_state=42)
dTree.fit(X_train, y_train)
Out[86]:
Scoring our Decision Tree¶
In [87]:
print(dTree.score(X_train, y_train))
print(dTree.score(X_test, y_test))
Visualizing the Decision Tree¶
In [88]:
train_char_label = ['No', 'Yes']
Credit_Tree_File = open('credit_tree.dot','w')
dot_data = tree.export_graphviz(dTree, out_file=Credit_Tree_File, feature_names = list(X_train), class_names = list(train_char_label))
Credit_Tree_File.close()
In [89]:
#Works only if "dot" command works on you machine
retCode = system("dot -Tpng credit_tree.dot -o credit_tree.png")
if(retCode>0):
print("system command returning error: "+str(retCode))
else:
display(Image("credit_tree.png"))
In [90]:
print_model_score(dTree, "Decision Tree")
df_compare_model
Out[90]:
In [91]:
print_model_roc(dTree)
In [92]:
visClassifierResults(dTree)
In [93]:
# Let's look at the feature importance
imp_features = pd.DataFrame(dTree.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by='Imp', ascending=False)
imp_features[imp_features.Imp >= 0.01].sort_values(by='Imp', ascending=False)
Out[93]:
Decision Tree reducing over fitting (Regularization) ¶
In [94]:
dTreeR = DecisionTreeClassifier(criterion = 'gini', max_depth = 6, random_state=42)
dTreeR.fit(X_train, y_train)
print(dTreeR.score(X_train, y_train))
print(dTreeR.score(X_test, y_test))
In [95]:
# train_char_label = ['No', 'Yes']
# Credit_Tree_FileR = open('credit_treeR.dot','w')
# dot_data = tree.export_graphviz(dTreeR, out_file=Credit_Tree_FileR, feature_names = list(X_train), class_names = list(train_char_label))
# Credit_Tree_FileR.close()
# #Works only if "dot" command works on you machine
# retCode = system("dot -Tpng credit_treeR.dot -o credit_treeR.png")
# if(retCode>0):
# print("system command returning error: "+str(retCode))
# else:
# display(Image("credit_treeR.png"))
In [96]:
# Draw Decision tree
draw_tree(dTreeR, X_train, y_train, 'dtree_model.dot', 'dtree_model.png', ['No', 'Yes'], list(X_train))
In [97]:
# importance of features in the tree building ( The importance of a feature is computed as the
# (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
# Let's look at the feature importance
imp_features = pd.DataFrame(dTreeR.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by='Imp', ascending=False)
imp_features[imp_features.Imp > 0.001].sort_values(by='Imp', ascending=False)
Out[97]:
In [98]:
print_model_score(dTreeR, "Decision Tree Regressor")
In [99]:
df_compare_model
Out[99]:
The confusion matrix
True Positives (TP): we correctly predicted that they will take personal loans 471
True Negatives (TN): we correctly predicted that they will not take personal loan 11709
False Positives (FP): we incorrectly predicted that they will take personal loan (a "Type I error") 257 Falsely predict positive Type I error
False Negatives (FN): we incorrectly predicted that they will not take personal loans (a "Type II error") 1127 Falsely predict negative Type II error
In [100]:
print_model_roc(dTreeR)
In [101]:
visClassifierResults(dTreeR)
Ensemble Learning - Bagging ¶
In [102]:
from sklearn.ensemble import BaggingClassifier
bgcl = BaggingClassifier(base_estimator=dTree, n_estimators=50,random_state=42)
#bgcl = BaggingClassifier(n_estimators=50,random_state=1)
bgcl = bgcl.fit(X_train, y_train)
# y_predict = bgcl.predict(X_test)
In [103]:
print_model_score(bgcl, "Bagging")
In [104]:
df_compare_model
Out[104]:
In [105]:
print_model_roc(bgcl)
In [106]:
visClassifierResults(bgcl)
Ensemble Learning - AdaBoosting ¶
In [107]:
from sklearn.ensemble import AdaBoostClassifier
abcl = AdaBoostClassifier(n_estimators=10, random_state=42)
#abcl = AdaBoostClassifier( n_estimators=50,random_state=1)
abcl = abcl.fit(X_train, y_train)
In [108]:
print_model_score(abcl, 'Adaptive Boosting')
df_compare_model
Out[108]:
In [109]:
print_model_roc(abcl)
In [110]:
visClassifierResults(abcl)
Ensemble Learning - GradientBoost ¶
In [111]:
from sklearn.ensemble import GradientBoostingClassifier
gbcl = GradientBoostingClassifier(n_estimators = 50,random_state=42)
gbcl = gbcl.fit(X_train, y_train)
In [112]:
print_model_score(gbcl, 'Gradient Boost')
In [113]:
print_model_roc(gbcl)
In [114]:
visClassifierResults(gbcl)
Ensemble RandomForest Classifier ¶
In [115]:
from sklearn.ensemble import RandomForestClassifier
rfcl = RandomForestClassifier(n_estimators = 100, random_state=42)
# rfcl = RandomForestClassifier(n_estimators = 50, random_state=1, max_features=12)
rfcl = rfcl.fit(X_train, y_train)
In [116]:
print_model_score(rfcl, "Random Forest")
In [117]:
print_model_roc(rfcl)
In [118]:
visClassifierResults(rfcl)
Make a DataFrame to compare models and their metrics. Give conclusion regarding the best algorithm and your reason behind it ¶
We have been creating the compare dataframes when running different models.¶
In [119]:
df_compare_model
Out[119]:
Random Forest and Bagging performed very close and much better in train and test score¶
Bagging did better in recall score 0.431 and Ramdom Forest is the next 0.388¶
Ramdom Forest did better in precision score than all others¶
Bagging did better in F1 score and ROC and the next best was Ramdom Forest¶
Predicted that customer who are interested in opening term deposit account will be approached
Here the bank wants to reach people who will open term deposit but our model predicted they will not take loan i.e. less number of False Negative, if FN is high, bank would lose on prospect customers. So that the bank doesn't lose money who are willing to open term deposit. Hence Recall is the important metric.
In case of False positive, bank will lose effort to reach out to a few people but thats okay because the bank thought that these people will open term deposit but they did not take. This number is quite low i.e. 288 for Random Forest and 426 for Bagging and the precision percent is quite good 68% for Random Forest as compared to 61% for Bagging.